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Why  GAO  Did  This  Study 

The  Joint  Exercise  Program  is  the 
principai  means  for  combatant 
commanders  to  maintain  trained  and 
ready  forces,  exercise  contingency 
and  theater  security  cooperation 
pians,  and  conduct  joint  and 
muitinationai  training  exercises. 
These  exercises  are  primariiy  aimed 
at  deveioping  the  skiiis  needed  by 
U.S.  forces  to  operate  in  a  joint 
environment  and  can  aiso  heip  buiid 
partner-nation  capacity  and 
strengthen  aiiiances. 

House  Report  114-102  inciuded  a 
provision  for  GAO  to  review  DOD’s 
Joint  Exercise  Program.  This  report 
(1)  describes  guidance  DOD  has 
deveioped  for  its  Joint  Exercise 
Program  and  DOD’s  approach  to 
assess  the  effectiveness  of  the 
program  and  (2)  evaiuates  the  extent 
to  which  DOD  uses  two  information 
technoiogy  systems  to  manage  the 
program.  GAO  observed  data  in 
JTIMS  and  anaiyzed  fiscai  years 
2014-16  financiai  data  and 
supporting  documentation  in  the 
Execution  Management  System. 

What  GAO  Recommends 

GAO  recommends  that  DOD  compiy 
with  current  guidance  to  upioad 
supporting  documentation  in  the 
Execution  Management  System  and 
impiement  effective  internai  controis 
to  ensure  the  compieteness  and 
accuracy  of  financiai  information. 
DOD  partiaiiy  concurred  with  both 
recommendations,  noting  existing 
controis  in  other  reiated  systems  of 
record.  GAO  beiieves  the 
recommendations  remain  vaiid,  as 
discussed  in  this  report. 

View  GAO-17-7.  For  more  information,  contact 
Cary  B.  Russell  at  (202)  512-5431  or 
russellc@gao.gov. 
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What  GAO  Found 

The  Department  of  Defense  (DOD)  has  deveioped  a  body  of  guidance  for  the 
Joint  Exercise  Program  and  has  impiemented  an  approach  to  assess  the 
effectiveness  of  the  program.  In  addition  to  the  body  of  guidance  for  the  program, 
DOD  is  working  to  update  a  key  guidance  document  for  miiitary  training  in 
accordance  with  a  congressionai  requirement.  DOD’s  approach  to  assess  the 
effectiveness  of  the  Joint  Exercise  Program  is  aimed  at  ensuring  that  its 
performance  measures  are  specific,  measurabie,  achievabie,  reaiistic,  and  time- 
phased  (commoniy  referred  to  as  the  SMART  rubric).  The  Joint  Assessment  and 
Enabiing  Capabiiity  office  reviews  the  performance  measures  created  by  the 
combatant  commands  against  this  rubric  and  provides  input  and  coaching  on 
improving  the  measures  through  an  ongoing  and  coiiaborative  process. 

DOD  uses  two  key  information  technoiogy  systems — the  Joint  Training 
Information  Management  System  (JTIMS)  and  the  Execution  Management 
System — to  manage  the  execution  of  the  Joint  Exercise  Program,  but  does  not 
have  assurance  that  funding  execution  data  in  the  Execution  Management 
System  are  reiiabie.  JTIMS  is  the  system  of  record  for  the  Joint  Exercise 
Program  that  combatant  commanders  use  to  pian  and  manage  their  joint  training 
exercises.  GAO  observed  significant  variation  in  the  type  and  quaiity  of 
information  entered  in  JTIMS.  Combatant  command  and  Joint  Staff  officiais 
stated  that  information  in  JTIMS  lacked  consistency  in  the  level  of  detail 
provided,  sometimes  making  it  difficult  to  coordinate  training  with  other 
combatant  commands  or  extract  pertinent  information  about  exercises  from  the 
system  that  would  be  helpful  in  planning  other  exercises.  Consequently,  to  help 
improve  the  consistency  and  standardization  of  information  across  combatant 
commands,  the  Joint  Staff  published  a  user  guide  for  JTIMS. 

Regarding  the  Execution  Management  System,  a  web-based  database  DCD 
uses  to  track  the  most  recent  funding  execution  data  for  the  Joint  Exercise 
Program,  GAC  found  that  DCD  does  not  have  assurance  that  the  system 
produces  quality  information  because  supporting  documentation  is  not 
consistently  uploaded  into  the  system  and,  when  it  is  uploaded,  it  is  not 
reconcilable  to  the  data  entered  there.  Gniy  U.S.  Strategic  Command  and  U.S. 
Northern  Command  uploaded  supporting  documentation  for  fiscal  years  2013-16 
as  required  by  the  Execution  Management  System  guidance.  Reviewing  a 
nongeneralizable  sample  of  uploaded  supporting  documentation  for  fiscal  years 
2014-16,  GAC  found  that  the  sum  of  the  individual  expenditures  reported  in 
supporting  documentation  did  not  match  corresponding  total  expenditures 
entered  in  the  system  for  any  of  the  four  combatant  commands  included  in 
GAC’s  review.  Further,  the  four  combatant  commands  GAG  visited,  the  Gffice  of 
the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  the  Gffice  of  the  Assistant 
Secretary  of  Defense  for  Readiness  had  not  implemented  effective  internal 
controls  similar  to  those  identified  in  the  Standards  for  Internal  Control  In  the 
Federal  Government  to  ensure  the  completeness  and  accuracy  of  financial 
information  captured  for  the  Joint  Exercise  Program.  Without  such  internal 
controls,  DCD  and  other  key  decision  makers  may  not  have  the  financial 
information  of  sufficient  quality  to  defend  the  Joint  Exercise  Program’s  budget. 
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Figure  1:  Exercise  and  Funding  Trends  for  the  Combatant 
Commanders  Exercise  Engagement  and  Training 
Transformation  Account,  fiscal  years  2013-16  6 

Figure  2:  Joint  Training  System  Used  by  Combatant  Commanders 
to  Execute  the  Department  of  Defense’s  Joint  Exercise 
Program  8 
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February  24,  2017 
Congressional  Committees 

The  Joint  Exercise  Program  is  a  principal  means  for  combatant 
commanders  to  maintain  trained  and  ready  forces,  exercise  contingency 
and  theater  security  cooperation  plans,''  and  conduct  joint  and 
multinational  training.  In  fiscal  year  201 6,  as  part  of  this  program,  the 
Department  of  Defense’s  (DOD)  combatant  commands^  participated  in 
more  than  150  joint  training  exercises'^  that  ranged  in  size  from  small- 
scale  events  to  major  joint  and  multilateral  exercises.  For  this  same 
period,  funding  for  these  exercises  was  approximately  $600  million. 
Although  these  exercises  have  the  primary  purpose  of  training  U.S. 
forces,  DOD  has  stated  they  can  also  help  build  partner-nation  capacity 
and  strengthen  alliances.  Furthermore,  joint  exercises  are  designed  to 
integrate  and  synchronize  interdependent  capabilities — such  as 
intelligence,  surveillance,  and  reconnaissance;  electronic  warfare;  and 
special  operations  forces — across  multiple  services  and  commands.  DOD 
officials  stated  that  participation  in  joint  exercises  enables  the  military 
services  to  build  trust  and  relationships  with  one  another,  U.S.  allies,  and 
potential  partners  while  developing  the  skills  necessary  to  operate  in  the 
joint  environment.  For  example,  in  fiscal  year  2015,  U.S.  Northern 
Command  (NORTHCOM)  conducted  a  joint  exercise  referred  to  as 
Ardent  Sentry  that  trained  federal,  state,  local,  and  tribal  governments 
and  private  and  nonprofit  organizations  on  how  to  respond  to  a  natural 
disaster.  DOD  uses  two  key  information  technology  systems  to  manage 
its  Joint  Exercise  Program. 


^Theater  Security  Cooperation  Plans  lay  out  strategies  for  working  with  other  countries  in 
a  region  to  further  U.S.  foreign  policy  and  military  goals. 

^DOD  has  nine  combatant  commands,  each  with  an  assigned  geographic  region  or 
function.  The  six  geographic  commands,  which  have  defined  areas  of  operation  and  have 
a  distinct  regional  military  focus,  are  U.S.  Africa  Command,  U.S.  Central  Command,  U.S. 
European  Command,  U.S.  Northern  Command,  U.S.  Pacific  Command,  and  U.S. 
Southern  Command.  The  three  functional  commands,  which  have  unique  capabilities  and 
operate  worldwide,  are  U.S.  Special  Operations  Command,  U.S.  Strategic  Command,  and 
U.S.  Transportation  Command.  Our  review  focused  on  U.S.  Northern  Command,  U.S. 
Pacific  Command,  U.S.  Strategic  Command,  and  U.S.  Transportation  Command. 

^“Joint  training”  is  military  training  based  on  joint  doctrine  designed  to  prepare  joint  forces 
and/or  joint  staffs  to  respond  to  operational  requirements  identified  by  joint  force 
commanders.  Joint  training  generally  involves  the  performance  of  tasks  by  members  or 
elements  of  more  than  one  military  service  under  the  direction  of  a  joint  force  commander. 
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House  Report  1 14-102,  accompanying  a  bill  for  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2016,  included  a  provision  for  us  to 
review  DOD’s  Joint  Exercise  Program.  Accordingly,  this  report  (1) 
describes  guidance  DOD  has  developed  for  its  Joint  Exercise  Program 
and  DOD’s  approach  to  assess  the  effectiveness  of  the  program  and  (2) 
evaluates  the  extent  to  which  DOD  uses  two  key  information  systems — 
the  Joint  Training  Information  Management  System  (JTIMS)  and  the 
Execution  Management  System-to  manage  its  Joint  Exercise  Program. 

To  describe  guidance  that  DOD  has  developed  for  its  Joint  Exercise 
Program  and  DOD’s  approach  to  assess  the  effectiveness  of  the 
program,  we  reviewed  guidance  developed  by  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Readiness,  the  Office  of  the  Chairman  of  the 
Joint  Chiefs  of  Staff,  and  the  combatant  commands  for  joint  exercises  and 
other  joint  training  events  funded  by  the  Combatant  Commanders 
Exercise  Engagement  and  Training  Transformation  account. We 
compared  guidance  issued  by  the  Office  of  the  Assistant  Secretary  of 
Defense  for  Readiness  with  that  issued  by  the  Office  of  the  Chairman  of 
the  Joint  Chiefs  of  Staff  to  identify  similarities  and  differences.  In  addition, 
we  reviewed  guidance  on  how  assessments  should  be  conducted  for  joint 
exercises.  Specifically,  we  reviewed  DOD’s  approach  for  making 
performance  measures  specific,  measurable,  achievable,  realistic,  and 
time-phased  (commonly  referred  to  as  the  SMART  rubric)  to  assess  the 
return  on  investment  for  the  Joint  Exercise  Program.  We  did  not  review  or 
evaluate  the  quality  of  any  assessments  that  DOD  has  conducted  for  its 
joint  exercises.  Further,  we  interviewed  senior  officials  from  the  Office  of 
the  Assistant  Secretary  of  Defense  for  Readiness,  including  the  Joint 
Assessment  and  Enabling  Capability  office,  the  Office  of  the  Chairman  of 
the  Joint  Chiefs  of  Staff,  as  well  as  officials  from  the  following  four 
combatant  commands  that  we  visited:  NORTHCOM,  U.S.  Pacific 
Command  (PACOM),  U.S.  Strategic  Command  (STRATCOM),  and  U.S. 
Transportation  Command  (TRANSCOM).  We  selected  these  four 
combatant  commands  for  our  site  visits  to  achieve  a  mix  of  geographical 
and  functional  commands.  In  addition  to  using  the  type  of  command  to 
make  our  selections,  we  also  selected  commands  based  on  the  funds 
that  had  been  apportioned  in  fiscal  year  2016  from  the  Combatant 


^The  Combatant  Commanders  Exercise  Engagement  and  Training  Transformation 
program  is  a  DOD-wide  joint  training  and  exercise  engagement  program  that  provides 
resources  for  the  combatant  commands  to  execute  joint  exercises  and  provides 
engagement  support  of  theater  campaign  plans,  objectives,  and  mission  essential  task- 
based  joint  training  initiatives. 
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Commanders  Exercise  Engagement  and  Training  Transformation  account 
to  those  combatant  commands. 

To  evaluate  DOD’s  use  of  JTIMS^  and  the  Execution  Management 
System®  to  manage  the  Joint  Exercise  Program,  we  reviewed  guidance 
for  JTIMS  and  the  Execution  Management  System;  observed  data 
associated  with  a  nongeneralizable  sample  of  joint  exercises  maintained 
in  JTIMS;  and  reviewed  and  analyzed  a  nongeneralizable  sample  of 
cumulative  financial  data  and  supporting  documentation,  if  any,  entered 
by  combatant  command  users  in  the  Execution  Management  System  for 
the  Joint  Exercise  Program  during  fiscal  years  2013-16  to  examine  the 
internal  controls  that  were  in  place. ^  We  reviewed  supporting 
documentation  uploaded  into  the  Execution  Management  System  to 
determine  whether  key  controls  were  in  place  to  ensure  that  the  system 
maintained  quality  information,  as  defined  in  the  Standards  for  Internal 
Control  in  the  Federal  Government.®  Additionally,  we  spoke  with 
cognizant  officials  from  the  four  combatant  commands  we  visited,  the 
Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  the  Office  of  the 
Assistant  Secretary  of  Defense  for  Readiness  about  the  systems  used  to 
manage  DOD’s  Joint  Exercise  Program.  We  also  reviewed  the  FIAR 
plan —  DOD’s  strategic  plan  and  management  tool  for  guiding, 
monitoring,  and  reporting  on  the  department’s  ongoing  financial 
management  improvement  efforts  and  for  communicating  the 


®JTIMS  is  the  system  of  record  used  by  the  combatant  commands  to  design,  plan, 
prepare,  execute,  and  evaluate  exercises. 

®The  Execution  Management  System  is  a  web-based  reporting  tool  that  provides  Joint 
Exercise  Program  execution  performance  data  to  stakeholders,  program  managers,  and 
others.  It  is  intended  to  provide  increased  financial  situational  awareness  of  how  funds  for 
the  Joint  Exercise  Program  are  spent. 

^According  to  a  DOD  official,  the  Combatant  Commanders  Exercise  Engagement  and 
Training  Transformation  enterprise  began  using  the  Execution  Management  System  in 
2008.  The  Execution  Management  System  allows  visibility  of  data  for  only  the  years  when 
users  are  able  to  input  activity.  Procurement  dollars  have  a  3-year  window  for  obligations 
and  a  fourth  year  for  expenditures.  Therefore,  the  earliest  year  that  is  visible  for  users  to 
input  activity  into  the  system  is  fiscal  year  201 3  since  procurement  funds  for  that  year 
were  available  to  obligate  through  fiscal  year  201 5  and  would  be  expended  throughout 
fiscal  year  2016. 

®See  GAO,  Standards  for  Internal  Control  In  the  Federal  Government,  GAO-14-704G, 
(Washington,  D.C.:  Sept.  2014).  Per  paragraph  13.05  of  these  standards,  “quality 
information”  is  appropriate,  current,  complete,  accurate,  accessible,  and  provided  on  a 
timely  basis.  Management  uses  quality  information  to  make  informed  decisions  and 
evaluate  an  entity’s  performance  in  achieving  key  objectives  and  addressing  risks. 
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department’s  approach  to  addressing  its  financial  management 
weaknesses  and  achieving  financial  statement  audit  readiness —  and 
guidance.  Appendix  I  discusses  our  objectives,  scope,  and  methodology 
in  greater  detail. 

We  conducted  this  performance  audit  from  June  2015  to  February  2017 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


DOD’s  Joint  Exercise  Program  provides  an  opportunity  for  combatant 
commanders  to  (1)  train  to  the  mission  capability  requirements  described 
in  the  Joint  Mission-Essential  Task  List®  and  (2)  support  theater  or  global 
security  cooperation  requirements  as  directed  in  theater  or  in  global 
campaign  plans.  All  nine  of  the  combatant  commands,  as  well  as  the  four 
military  services,  conduct  exercises  as  a  part  of  the  Joint  Exercise 
Program.  The  mission  for  the  four  combatant  commands  we  visited  are 
as  follows: 

•  NORTHCOM  conducts  homeland  defense,  civil  support,  and  security 
cooperation  to  defend  and  secure  the  United  States  and  its  interests. 

•  PACOM,  with  assistance  from  other  U.S.  government  agencies, 
protects  and  defends  the  United  States,  its  people,  and  its  interests.  In 
conjunction  with  its  allies  and  partners,  PACOM’s  goal  is  to  enhance 
stability  in  the  Indo-Asia-Pacific  region  by  promoting  security 
cooperation,  responding  to  contingencies,  deterring  aggression,  and 
when  necessary,  fighting  to  win. 

•  STRATCOM  conducts  global  operations  in  coordination  with  other 
combatant  commands,  military  services,  and  appropriate  U.S. 
government  agencies  to  deter  and  detect  strategic  attacks  against  the 
United  States,  its  allies,  and  partners. 


®The  Joint  Mission  Essential  Task  List  is  a  joint  force  commander’s  list  of  priority  tasks 
derived  from  plans  and  orders  along  with  associated  conditions  and  measurable 
standards.  It  constitutes  the  joint  force  commander’s  warfighting  requirements  based  on 
assigned  missions.  The  joint  mission  essential  tasks  are  those  identified  by  the  joint  force 
commander  as  being  the  most  essential  to  the  assigned  or  anticipated  missions,  with 
priority  given  to  wartime  missions. 
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•  TRANSCOM  provides  a  full-spectrum  of  global  mobility  solutions  and 
related  enabling  capabilities  for  supported  customers’  requirements  in 
peace  and  war. 

The  key  players  with  roles  and  responsibilities  in  the  Joint  Exercise 
Program  are  as  follows: 

•  Principal  Deputy  Assistant  Secretary  of  Defense  for  Readiness, 
whose  responsibilities  include  administering  the  Combatant 
Commanders  Exercise  Engagement  and  Training  Transformation 
account; 

•  Director  for  Joint  Force  Development  Joint  Staff  (J7),  whose 
responsibilities  include  managing  the  Combatant  Commanders 
Exercise  Engagement  and  Training  Transformation  account  and 
providing  enabling  capabilities  that  support  combatant  commands’ 
and  the  military  services’  training; 

•  combatant  commands,  who  develop,  publish,  and  execute  command 
Joint  Training  Plans^°  and  joint  training  programs  for  command  staff 
and  assigned  forces;  and 

•  military  services,  whose  responsibilities  include  providing  trained  and 
ready  forces  for  joint  employment  and  assignment  to  combatant 
commands. 

In  fiscal  year  2016,  the  Combatant  Commanders  Exercise  Engagement 
and  Training  Transformation  account  provided  approximately  $600  million 
dollars  to  fund  more  than  150  training  events.  Funding  from  this  account 
covers  items  such  as  personnel  travel  and  per  diem  for  planning 
conferences  and  exercise  support  events,  transportation  of  cargo,  airlift, 
sealift  and  port  handling,  intra-theater  transportation  for  participating 
units,  consultant  advisory  and  assistance  service,  equipment  and 
supplies,  and  operation  and  maintenance  for  training  support  facilities  and 
equipment.  From  fiscal  year  201 3  through  fiscal  year  201 6,  funding  for 
this  account  decreased  by  about  $149  million,  or  by  20  percent,  while  the 
number  of  exercises  conducted  remained  relatively  unchanged  (see  fig. 

1). 


Joint  Training  Plan  is  developed  and  updated  annually  by  each  joint  commander  to 
document  the  Joint  Training  Strategy  that  will  be  used  for  training  assigned  forces  to 
accomplish  the  mission  requirements. 
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Figure  1:  Exercise  and  Funding  Trends  for  the  Combatant  Commanders  Exercise 
Engagement  and  Training  Transformation  Account,  fiscai  years  2013-16 


Number  of  □  —Funding 

exercises  (miilions  of  doiiars) 

200  800 


Fiscai  year 

Source:  GAO  analysis  of  Department  of  Defense  data.  I  GAO-17-7 


DOD  officials  told  us  that  in  part  as  a  result  of  reduced  funding  for  the 
Joint  Exercise  Program  they  have  at  times  reduced  the  scope  of 
exercises  or  sought  alternative  methods,  such  as  relying  on  organic  lift 
capabilities  of  Service  components  or  partnering  with  another  combatant 
command  to  execute  exercises.  Other  factors  that  could  impact  the  ability 
of  combatant  commands  to  execute  exercises  include  the  availability  of 
forces;  diplomatic  (political  and  military)  considerations;  and  real  world 
events,  such  as  natural  disasters.  Though  DOD  officials  stated  that  these 
factors  are  largely  outside  of  the  sphere  of  combatant  commander 
influence  and  therefore  are  not  included  in  the  original  planning  of  the 
exercises,  officials  stated  that  they  use  various  approaches  to  try  to 
mitigate  the  effect  these  factors  have  on  their  ability  to  carry  out  their 
respective  joint  exercise  programs.  If  these  factors  cannot  be  mitigated, 
the  combatant  command  might  cancel  a  joint  exercise,  a  mitigation 
strategy  of  last  resort. 
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DOD  Has  Developed 
Guidance  for  the  Joint 
Exercise  Program 
and  Has  Implemented 
an  Approach  to 
Assess  Program 
Effectiveness 


DOD  Has  Developed  a 
Body  of  Guidance  for  the 
Joint  Exercise  Program 
and  Is  Working  to  Update 
a  Key  Guidance 
Document  That  Is 
Outdated 

•  DODD  1322.18,  Military  Training,  (Jan.  13,  2009),  is  the  overarching 
guidance  for  military  training  and  identifies  the  roles  and 
responsibilities  for  training  military  individuals;  units;  DOD  civilian 
employees;  and  contractors,  among  others. 

•  The  Program  Goals  and  Objectives  document  provides  guidance  for 
all  programs  and  activities  that  utilize  funds  from  the  Combatant 
Commanders  Exercise  Engagement  and  Training  Transformation 
account. 

•  CJCSI  3500.01  H,  Joint  Training  Policy  for  the  Armed  Forces  of  the 
United  States,  (April  25,  2014),  establishes  guidance  for  the  Joint 
Training  System — an  integrated,  requirements-based,  four-phased 
approach  that  is  used  to  align  a  combatant  commander’s  Joint 
Training  Strategy  with  assigned  missions  to  produce  trained  and 
ready  individuals,  staff,  and  units.  The  Joint  Training  System  is  used 
by  combatant  commanders  to  execute  the  Joint  Exercise  Program  as 
shown  in  figure  2.  See  appendix  II  for  a  more  detailed  description  of 
the  Joint  T raining  System. 

^^H.R.  Rep.  No.  114-537,  accompanying  H.R.  4909,  National  Defense  Authorization  Act 

for  Fiscal  Year  2017  (May  4,  2016). 


DOD  has  developed  a  body  of  guidance  for  the  Joint  Exercise  Program 
and  is  working  to  update  a  key  outdated  guidance  document  that 
identifies  overarching  roles  and  responsibilities  for  military  training  in 
accordance  with  a  congressional  requirement  in  a  House  Committee  on 
Armed  Services  report  accompanying  the  National  Defense  Authorization 
Act  for  Fiscal  Year  2017.'’''  DOD-wide  guidance,  policies,  and  procedures 
addressing  various  aspects  of  the  Joint  Exercise  Program  are  contained 
in  the  following  documents: 
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Figure  2:  Joint  Training  System  Used  by  Combatant  Commanders  to  Execute  the 
Department  of  Defense’s  Joint  Exercise  Program 


Phase  I:  Requirements 


Derived  from  guidance  such  as  the 
Unified  Command  Plan,  Guidance  for 
Employment  of  the  Force,  the  Joint 
Strategic  Capabilities  Plan,  and  aligned 
missions 

•  Based  on  commanders'  intent  and  joint 
doctrine 

•  Stated  in  terms  of  joint  mission- 
essential  tasks,  conditions,  and 
standards  entered  into  the  Defense 
Readiness  Reporting  System® 


Phase  IV:  Assessment 


Commander/director  assesses: 

•  Training  effectiveness 

•  Joint  readiness  to  perform  organization 
mission 

Joint  Training  Information  Management 
System  training  data  are  input  in  Defense 
Readiness  Reporting  System 


Driven  by  mission  requirements,  the 
training  audience,  and  the  tools  available 

•  Creates  the  Joint  Training  Plan^  and 
establishes  objectives  to  meet 
requirements 

•  Guidance  provided  and  schedules 
deconfiicted  using  the  Joint  Training 
Information  Management  System'^ 


Phase  III:  Execution 


Joint  training  conducted  by  the  Office  of 
the  Chairman  of  the  Joint  Chiefs  of  Staff, 
the  combatant  commands,  combat 
support  agencies,  the  National  Guard 
Bureau,  and  the  military  services 

•  Joint  training  events  executed,  training 
audience  performance  objectives 
observed,  and  performance  outcomes 
evaluated 

•  Commander/director  responsible  for 
ensuring  performance  in  all  training 
events  is  evaluated 


Source:  GAO  analysis  of  Department  of  Defense  information.  |  GAO-17-7 


^The  Defense  Readiness  Reporting  System  is  used  by  the  Department  of  Defense  to  monitor  the 
readiness  of  the  department’s  components  to  provide  capabilities  to  support  the  National  Military 
Strategy  consistent  with  the  department’s  priorities  and  planning  direction.  Through  this  system, 
commanders,  military  service  chiefs,  and  agency  directors  assess  the  ability  of  their  respective 
organizations  to  accomplish  a  task  to  standards  under  conditions  specified  in  their  Joint  Mission- 
Essential  Task  List  or  Agency  Mission-Essential  Task  List. 

‘’In  the  Plans  phase,  the  Joint  Training  Plan  is  developed  and  updated  annually  by  each  joint 
commander  to  document  the  Joint  Training  Strategy  that  will  be  used  for  training  assigned  forces  to 
accomplish  the  mission  requirements. 

“The  Joint  Training  Information  Management  System  is  the  system  of  record  used  by  the  combatant 
commands  to  design,  plan,  prepare,  execute,  and  evaluate  exercises. 


•  CJCSN  3500.01 , 2015-2018  Chairman’s  Joint  Training  Guidance, 
(Oct.  30,  2014),  provides  the  Office  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff’s  joint  training  guidance  to  all  DOD  components  for  the 
planning,  execution,  and  assessment  of  joint  individual  and  collective 
training  for  fiscal  years  2015  through  2018. 
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•  CJCSM  3500. 03E,  Joint  T raining  Manual  for  the  Armed  Forces  of  the 
United  States,  (April  20,  2015),  provides  guidance  and  procedures  for 
the  Joint  Training  System.  Specifically,  it  focuses  on  determining  joint 
training  requirements,  planning  and  executing  joint  training,  and 
assessing  joint  training. 

•  CJCSM  351 1.01,  Joint  T raining  Resources  of  the  Armed  Forces  of  the 
United  States,  (May  26,  2015),  provides  detailed  guidance  on  joint 
funding,  joint  transportation,  and  joint  training  support  resources  for 
joint  training  exercises. 

•  CJCSI  3150.25F,  Joint  Lessons  Learned  Program,  (June  26,  2015), 
provides  guidance  for  gathering,  developing,  and  disseminating  joint 
lessons  learned  for  the  armed  forces  for  joint  training  exercises. 

Each  of  the  combatant  commands  we  visited  had  developed  their  own 
implementation  guidance,  which  is  consistent  with  DOD’s  guidance  for 
the  Joint  Exercise  Program. 

While  DOD  has  a  body  of  guidance  for  the  Joint  Exercise  Program, 

DODD  1322.18  -  key  overarching  guidance  for  military  training  that 
identifies  roles  and  responsibilities  for  training,  including  the  for  the  Joint 
Exercise  Program-is  outdated.  Specifically,  this  directive  assigns 
significant  roles  and  responsibilities  relevant  to  the  Joint  Exercise 
Program  to  U.S.  Joint  Forces  Command,''^  a  combatant  command  that 
has  not  existed  since  August  2011.  For  example,  according  to  DODD 
1322.18,  U.S.  Joint  Forces  Command  is  responsible  for  working  through 
the  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff  to  manage  joint 
force  training,  accredit  joint  training  programs  for  designated  joint  tasks, 
and  provide  Combatant  Commanders  Exercise  Engagement  and  Training 
Transformation  funds  to  support  the  Joint  Exercise  Program.  According  to 
subsequent  guidance  issued  in  April  2014,^^  the  Office  of  the  Chairman  of 
the  Joint  Chiefs  of  Staff  was  assigned  the  roles  and  responsibilities 
formerly  performed  by  U.S.  Joint  Forces  Command.  Additionally,  the 
Office  of  the  Assistant  Secretary  of  Defense  for  Readiness,  instead  of 
U.S.  Joint  Forces  Command,  now  administers  the  Combatant 


^^U.S.  Joint  Forces  Command  was  one  of  DOD’s  combatant  commands  that  had  multiple 
missions  including  to  serve  as  joint  force  provider  and  trainer  for  homeland  defense,  until 
being  de-established  in  201 1 . 

^^Chairman  of  the  Joint  Chiefs  of  Staff  Instruction,  CJCSI  3500.01  H,  Joint  Training  Poiicy 
for  the  Armed  Forces  of  the  United  States,  (April  25,  2014). 
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Commanders  Exercise  Engagement  and  Training  Transformation 
account,  which  funds  the  Joint  Exercise  Program. 

In  House  Report  1 14-537,^'^  the  House  Committee  on  Armed  Services 
also  noted  that  DODD  1322.18  is  outdated  and  does  not  account  for 
significant  organizational  changes  that  have  occurred  within  the 
department — specifically,  the  disestablishment  of  U.S.  Joint  Forces 
Command  and  the  establishment  of  the  Assistant  Secretary  of  Defense 
for  Readiness.  Consequently,  the  report  directs  DOD  to  update  its 
guidance  and  brief  the  committee  on  its  progress  updating  the  guidance 
by  December  1,  2016.  According  to  an  official  from  Office  of  the  Assistant 
Secretary  of  Defense  for  Readiness,  the  office  responsible  for  DODD 
1322.18,  the  department  is  aware  that  the  directive  is  outdated  and  is 
working  on  updating  it  but  is  unsure  of  when  the  update  process  will  be 
completed.  Specifically,  according  to  this  DOD  official,  the  department  is 
working  to  determine  whether  the  directive  can  be  updated  through  an 
administrative  update,  which  requires  less  coordination  and  time  to 
process  than  doing  so  through  a  total  reissuance  of  guidance.  When  DOD 
completes  the  update  and  includes  information  on  current  roles  and 
responsibilities,  the  key  guidance  regarding  the  Joint  Exercise  Program 
should  be  consistent  with  other  guidance. 


DOD  Has  Implemented  an 
Approach  Intended  to 
Assess  the  Effectiveness 
of  the  Joint  Exercise 
Program 


DOD  has  implemented  an  approach  to  assess  the  return  on  investment 
for  the  Joint  Exercise  Program.''®  The  Director  of  the  Joint  Assessment 
and  Enabling  Capability  office  stated  that  officials  from  that  office  provide 
the  combatant  commands  with  guidance  for  how  to  develop  performance 
measures  to  assess  the  effectiveness  of  the  Joint  Exercise  Program. 
Specifically,  the  combatant  commands,  in  conjunction  with  this  office, 
develop  performance  measures  using  an  approach  that  is  aimed  at 
ensuring  the  performance  measures  are  specific,  measurable, 
achievable,  realistic,  and  time-phased  (commonly  referred  to  as  the 
SMART  rubric).''®  The  Director  of  the  Joint  Assessment  and  Enabling 
Capability  office  reviews  the  performance  measures  created  by  the 


^"'h.R.  Rep.  No.  114-537,  accompanying  H.R.  4909,  National  Defense  Authorization  Act 
for  Fiscal  Year  2017  (May  4,  2016). 

^®We  did  not  review  or  evaluate  the  quality  of  DOD’s  assessments  of  the  Joint  Exercise 
Program. 

^®The  combatant  commands  also  use  the  SMART  rubric  to  develop  performance 
measures  to  assess  individual  joint  exercises. 
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combatant  commands  against  the  SMART  rubric  and  provides  input  and 
coaching  on  improving  the  measures  through  an  ongoing  and 
collaborative  process.  See  appendix  III  for  a  more  detailed  explanation  of 
DOD’s  approach  for  assessing  individual  joint  exercises. 

The  Joint  Assessment  and  Enabling  Capability  office  is  working  with 
individual  combatant  commands  to  develop  measures  to  assess  the 
return  on  investment  of  the  Joint  Exercise  Program  using  the  SMART 
rubric  approach.  For  example,  NORTHCOM  officials  told  us  that  they  are 
working  with  the  Joint  Assessment  and  Enabling  Capability  office  to 
develop  a  better  method  to  measure  the  return  on  investment  for 
NORTHCOM  joint  exercises  because  the  ones  they  currently  use,  such 
as  the  number  of  joint  mission-essential  tasks  in  an  exercise,  do  not 
reveal  any  information  that  would  be  helpful  for  decision  making.  The 
officials  stated  that  they  are  still  trying  to  determine  the  threshold  for  the 
amount  of  information  that  is  necessary  to  measure  return  on  investment 
for  their  joint  exercises.  Further,  officials  stated  that  they  were  drafting 
performance  measures  to  assess  return  on  investment  to  submit  to  their 
leadership  for  approval.  Additionally,  TRANSCOM  officials  told  us  that 
they  too  are  working  with  the  Joint  Assessment  and  Enabling  Capability 
office  but  have  not  yet  determined  how  to  effectively  gauge  return  on 
investment  for  training  dollars  spent  on  its  exercises. 

According  to  DOD  and  combatant  command  officials  we  interviewed, 
readiness  is  their  key  performance  measure  and  they  have  ongoing 
efforts  to  develop  more  tangible,  quantifiable  measures  to  determine  the 
return  on  investment  for  conducting  exercises.  However,  according  to 
combatant  command  officials,  return  on  investment  is  sometimes 
intangible  and  may  not  be  seen  immediately.  Officials  stated  that  it  could 
take  years  to  recognize  the  return  on  investment  for  conducting  an 
exercise.  For  example,  PACOM  officials  told  us  that  they  conducted  two 
multinational  planning  exercises  and  a  multinational  force  standard 
operating  procedures  workshop  designed  to  increase  the  speed  of  initial 
response  forces  to  an  emergent  issue  and  enhance  relationships  with 
partner  countries  for  several  years.  According  to  PACOM  officials,  the 
return  on  that  investment,  however,  was  not  realized  until  April  25,  2015, 
when  a  region  northwest  of  Kathmandu,  Nepal,  was  devastated  by  a  7.8 
magnitude  earthquake  and  the  after-action  report  for  that  earthquake 
indicated  that  PACOM’s  exercises  were  vital  in  preparing  the  Nepal  Army 
for  its  response. 
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DOD  Uses  Two  Key 
Information 
Technology  Systems 
to  Manage  the  Joint 
Exercise  Program, 
but  the  Data  in  the 
System  That  Tracks 
the  Most  Recent 
Funding  Execution 
Data  Does  Not  Have 
Required  Supporting 
Documentation  and 
Its  Data  Quality  Is 
Questionable 


DOD  uses  two  key  information  technology  systems — JTIMS  and  the 
Execution  Management  System — to  manage  the  execution  of  the  Joint 
Exercise  Program,  but  DOD  does  not  have  assurance  that  the  Execution 
Management  System  produces  quality  information.''^ 


DOD  Uses  JTIMS  to  Plan 
and  Manage  Training  and 
Has  Developed  a  User 
Guide  to  Standardize 
Information 


DOD  uses  JTIMS  as  the  system  of  record  for  the  Joint  Exercise  Program 
and  combatant  commanders  plan  and  manage  their  joint  training 
exercises  through  JTIMS.''®  JTIMS  automates  the  management  of  joint 
exercise  training  data  through  a  web-based  system  and  supports  the 
application  of  the  four  phases  of  the  Joint  Training  System.  Specifically, 
JTIMS,  which  is  managed  by  the  Office  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff,  is  used,  among  other  things,  to  (1)  request  and  track 
forces  for  joint  training  exercises,  (2)  publish  the  Joint  Training  Plan,  (3) 
document  and  manage  joint  training  programs,  and  (4)  capture  the 
assessments  of  exercises.  Chairman  of  the  Joint  Chiefs  of  Staff  policy 
requires  the  use  of  JTIMS  for  a  number  of  fields.  For  example,  guidance 
requires,  among  other  things,  that  the  combatant  commands  enter  key 
information  about  a  training  exercise,  such  as  its  objectives,  intended 


'^In  this  report  we  use  “quality  information”  and  “quality  data”  to  refer  to  information  or 
data  that  is  appropriate,  current,  complete,  accurate,  accessible,  and  provided  on  a  timely 
basis,  per  paragraph  1 3.05  of  GAO,  Standards  for  Internal  Control  In  the  Federal 
Government,  GAO-14-704G,  (Washington,  D.C.:  Sept.  2014). 

'®CJCSI  3500.01  H,  Joint  Training  Policy  for  the  Armed  Forces  of  the  United  States,  (April 
25,  2014). 
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audience  (i.e.,  the  joint  forces  being  trained),  lessons  learned,  and 
observations  on  performance,  and  costs.  However,  the  extent  to  which 
these  fields  are  used,  and  the  quality  of  the  data  entered  varies  by 
combatant  command.  Combatant  commands  and  other  DOD  entities  rely 
on  the  information  entered  in  JTIMS  to  both  conduct  their  exercises  and 
participate  in  exercises  sponsored  by  other  combatant  commands. 

However,  during  the  course  of  our  review,  we  were  informed  of  and 
observed  significant  variation  in  the  type  and  quality  of  information 
entered  in  JTIMS.  For  example,  officials  from  one  combatant  command 
stated  that  an  exercise  description  entered  by  another  combatant 
command  did  not  provide  sufficient  detail,  therefore  making  it  difficult  to 
understand  the  focus  of  the  exercise.  In  addition,  TRANSCOM  officials 
randomly  selected  exercises  in  JTIMS  to  show  us  the  type  of  information 
entered  in  the  system  and  we  noted  that  the  level  of  detail  provided 
sometimes  varied  significantly  by  combatant  command.  Furthermore, 
officials  from  two  of  the  four  combatant  commands  we  visited  stated  that 
sometimes  the  information  captured  in  JTIMS  is  not  useful  and  could 
negatively  affect  their  ability  to  coordinate  training  with  other  combatant 
commands  or  extract  pertinent  information  about  exercises  from  the 
system  that  would  be  helpful  in  planning  them. 

According  to  an  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff  official, 
it  is  important  that  combatant  commands  enter  information  in  JTIMS  in  a 
consistent  and  standardized  manner  so  that  the  information  is  easily 
understood  and  useful  for  all  joint  exercise  training  participants  and 
planners.  The  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff  and 
combatant  command  officials  told  us  that  the  lack  of  standardized 
information  in  JTIMS  is  due  to  the  absence  of  detailed  instructions  in 
guidance  on  inputting  information  into  JTIMS.  Consequently,  to  help 
improve  the  consistency  and  standardization  of  information  across 
combatant  commands,  the  Office  of  the  Chairman  of  the  Joint  Chiefs  of 
Staff  published  a  user  guide  for  JTIMS  that  is  intended  to  mitigate 
inconsistencies  in  the  information  entered  there,  standardize  the  use  of 
the  system  across  DOD,  and  improve  the  overall  understanding  of  the 
system.  According  to  this  official,  the  user  guide  was  completed  in 
October  2016.  The  Office  of  the  Chairman  of  the  Joint  Chiefs  Staff  plans 
to  periodically  update  the  user  guide  to  keep  pace  with  joint  training  policy 
updates,  JTIMS  software  upgrades,  and  joint  training  enterprise  business 
rule  modifications.  In  addition  to  providing  step-by-step  instructions  on 
using  JTIMS,  the  guide  also  provides  examples  of  the  type  of  information 
that  should  be  entered  in  specified  fields.  Such  information  should  help 
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improve  the  overall  understanding  of  and  bring  consistency  to  the  use  of 
JTIMS  across  the  combatant  commands. 


DOD  Uses  the  Execution 
Management  System  to 
Oversee  and  Report  the 
Most  Recent  Funding 
Execution  Data,  but  Data 
Quality  Has  Been  Lacking 
Due  to  Weaknesses  in 
Internal  Controls 


DOD  uses  the  Execution  Management  System,  a  web-based  database, 
to  track  and  oversee  the  most  recent  execution  performance  data,^® 
hereafter  referred  to  as  data,  for  the  Joint  Exercise  Program.  The 
Execution  Management  System  is  intended  to  capture  the  most  recent 
data  for  the  Joint  Exercise  Program.  According  to  officials,  it  is  important 
to  have  accurate  and  current  data  in  the  Execution  Management  System 
because  it  provides  instant  status  of  the  over-  and  underexecution  of 
funds  for  the  Joint  Exercise  Program,  which  is  critical  to  the  efficient  and 
effective  execution  of  the  Joint  Exercise  Program.  Moreover,  officials  from 
the  Assistant  Secretary  of  Defense  for  Readiness  office  stated  that  data 
from  this  system  are  used  to  report  how  funds  are  being  expended  for  the 
Joint  Exercise  Program  to  both  DOD  decision  makers  and  Congress. 


In  April  2016,  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Readiness  issued  guidance  on  the  use  of  the  Execution  Management 
System  to  the  combatant  commands.  This  guidance,  referred  to  as  the 
Execution  Management  System  Standard  Operating  Procedure  and  User 
Guide,  states  that  Joint  Exercise  Program  managers  are  required  to  (1) 
enter  the  most  recent  obligation  and  expenditure  amounts  for  any 
transactions  funded  through  the  Combatant  Commanders  Exercise 
Engagement  and  Training  Transformation  account  on  a  monthly  basis 
and  (2)  upload  supporting  documentation  for  transactions.  The  guide  also 
specifies  the  type  of  supporting  documents  that  should  be  uploaded  into 
the  Execution  Management  System,  such  as  awarded  contracts,  invoices, 
and  travel  payments.  Prior  to  issuing  guidance  in  April  2016,  an  official 
responsible  for  administering  the  Combatant  Commanders  Exercise 
Engagement  and  Training  Transformation  account  stated  that  the 
combatant  commands  were  informed  of  the  requirement  to  upload 
supporting  documentation  into  the  Execution  Management  System  in 
2011.  NORTHCOM,  STRATCOM,  and  PACOM  officials  told  us  that  they 
were  aware  of  this  requirement  prior  to  April  2016.  TRANSCOM  officials 
initially  stated  that  they  were  unaware  of  the  requirement;  however,  our 
review  of  the  Execution  Management  System  revealed  that  they  were 
uploading  supporting  documentation  for  some  fiscal  years  prior  to  the 
guidance  being  issued. 


^®For  the  purposes  of  this  report,  “execution  performance  data”  are  the  fiscal 
commitments,  obligations,  and  expenditures  data  that  are  entered  into  the  Execution 
Management  System  by  combatant  command  officials. 
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During  our  review  of  the  Execution  Management  System,  we  found  that 
the  combatant  commands  we  visited  had  not  fully  implemented  the 
guidance  and  that  the  quality  of  information  in  the  system  was 
questionable.  Specifically,  we  found  that: 

•  The  Execution  Management  System  is  missing  supporting 
documentation.  Based  on  our  review,  we  found  that  two — 
STRATCOM  and  NORTHCOM — of  the  four  combatant  commands  we 
visited  had  uploaded  supporting  documentation,  as  required  by  the 
Execution  Management  System  guidance,  for  fiscal  years  2013-16.  A 
third  combatant  command,  TRANSCOM,  uploaded  supporting 
documentation  for  fiscal  years  2013,  2014,  and  2016,  but  did  not 
upload  supporting  documentation  for  fiscal  year  201 5.  The  fourth 
combatant  command,  PACOM,  did  not  upload  supporting 
documentation  for  fiscal  years  2013,  2014,  and  2015,  but  began 
uploading  supporting  documentation  in  August  2016  for  fiscal  year 
2016  after  we  informed  an  official  in  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Readiness  that  the  command  had  not  been 
uploading  supporting  documentation  in  accordance  with  the  Execution 
Management  System  guidance.  TRANSCOM  and  PACOM  officials 
stated  that  one  of  the  reasons  they  did  not  upload  supporting 
documentation,  as  required  by  guidance,  was  due  to  the  volume  of 
travel  and  other  related  documents  generated  in  executing  joint 
training  exercises.  Officials  stated  that  it  was  overly  burdensome  to 
upload  all  of  these  documents.  Nonetheless,  an  official  from  the  Office 
of  the  Assistant  Secretary  of  Defense  for  Readiness  stated  the 
combatant  commands  need  to  do  their  due  diligence  in  uploading 
supporting  documentation  in  order  to  ensure  proper  accountability  of 
Combatant  Commanders  Exercise  Engagement  and  Training 
Transformation  funds.  This  official  further  stated  that  efforts  are 
underway,  that  includes  establishing  a  new  method  for  how  funds  are 
distributed  to  stakeholders,  to  identify  approaches  that  will  reduce  the 
data  entry  burden  at  the  stakeholder  level. 

•  Documentation  for  expenditures^”  upioaded  into  the  Execution 
Management  System  did  not  match  reported  totai  expenditures 
for  any  of  the  four  combatant  commands  we  visited.  Based  on  our 


^°While  the  Execution  Management  System  tracks  obligation,  expenditure,  and 
commitment  fiscal  data,  for  the  purposes  of  this  report,  we  use  the  term  “expenditures”  to 
generally  refer  to  all  the  data  entered  and  tracked  in  the  system.  Furthermore,  when  we 
attempted  to  link  the  sum  of  individual  transactions  with  cumulative  total  fields  in  the 
Execution  Management  System,  we  were  attempting  to  link  the  same  type  of 
transaction — ^for  example,  obligation  to  obligation,  and  commitment  to  commitment. 


Page  15 


GAO-17-7  Joint  Exercise  Program 


review  of  a  nongeneralizable  sample^^  of  supporting  documentation 
for  fiscal  years  2014  through  2016  that  was  uploaded  into  the 
Execution  Management  System,  we  found  that  the  sum  of  the 
individual  expenditures  reported  in  supporting  documentation  did  not 
match  the  corresponding  total  expenditures  entered  in  the  system  for 
any  of  the  four  combatant  commands  we  visited.  According  to  one 
combatant  command  official  familiar  with  this  system,  individual 
expenditures  reported  in  supporting  documents  should  be 
reconcilable  to  yearly  cumulative  totals  for  expenditures.  However, 
when  we  attempted  to  link  the  sum  of  individual  expenditures  reported 
in  uploaded  supporting  documentation  to  total  expenditures  data 
entered  into  the  Execution  Management  System  by  combatant 
command  officials,  we  were  unable  to  do  so  for  three  of  the  four 
combatant  commands  we  visited. For  example,  in  fiscal  year  201 5, 
NORTHCOM  uploaded  more  than  100  documents  that  supported  how 
funds  were  obligated  or  committed.  Our  review  found  that  the 
uploaded  documentation  supported  approximately  $12.7  million  in 
funds  that  were  committed.  However,  the  figure  entered  in  the 
Execution  Management  System  was  about  $1 1 .9  million.  Similarly,  in 
fiscal  year  2014,  TRANSCOM  supporting  documentation  showed  that 
commitments  totaled  approximately  $66.8  million  while  the  figure 
entered  in  the  Execution  Management  System  was  approximately 
$4.6  million.  Officials  stated  that  the  reason  that  the  supporting 
documentation  does  not  match  the  figures  entered  in  the  Execution 
Management  System  is  that  some  supporting  documentation  had  not 
been  uploaded.  Nonetheless,  the  inability  to  reconcile  supporting 
documentation  with  the  expenditures  entered  in  the  Execution 
Management  System  undermines  the  quality  of  the  data  in  the  system 
and  inhibits  DOD  decision  makers,  particularly  those  in  the  Office  of 
the  Assistant  Secretary  of  Defense  for  Readiness,  from  providing 
adequate  oversight  of  how  funds  are  being  expended  in  support  of  the 
Joint  Exercise  Program  goals.  Moreover,  the  inconsistent  uploading  of 


reviewed  all  supporting  documentation  for  fiscal  year  2014  for  TRANSCOM;  fiscal 
year  2015  for  NORTHCOM;  and  fiscal  year  2016  for  PACOM.  We  did  not  attempt  to 
reconcile  supporting  documentation  with  transaction  data  in  the  Execution  Management 
System  for  STRATCOM  because,  unlike  the  other  three  combatant  commands, 
STRATCOM  only  uploaded  its  monthly  execution  reports,  but  had  not  uploaded  individual 
supporting  documents  into  the  system,  such  as  awarded  contracts,  travel  payments,  and 
purchase  requests. 

^^Many  expenditures  do  not  occur  until  2  to  5  years  after  they  have  been  obligated.  The 
Execution  Management  System  does  not  allow  users  to  input  expenditures  for  previous 
months.  Therefore,  expenditures  that  occur  with  previous  fiscal  year  dollars  are  not  able  to 
be  reconciled  with  current  documents  in  the  Execution  Management  System. 
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the  required  supporting  documentation  and  difficulty  in  reconciling 
individually  reported  transactions  with  cumulative  values  entered  into 
the  system  suggests  that  weaknesses  exist  in  the  Execution 
Management  System  data  entry  procedures,  which  impacts  the 
quality  of  the  data  entered  in  the  system.  Therefore,  it  calls  into 
question  the  use  of  the  Engagement  Management  System  which, 
according  to  DOD  officials,  had  been  established  to  provide  real-time, 
accurate  information  on  the  execution  of  Joint  Exercise  Program 
funds  to  decision  makers. 

•  DOD  has  not  implemented  key  processes  to  help  ensure  that  the 
Execution  Management  System  produces  quality  information. 

Further  weakening  the  quality  of  the  reporting,  tracking,  and 
reconciliation  of  data  recorded  in  the  Execution  Management  System 
is  that  none  of  the  four  combatant  commands  we  visited,  the  Office  of 
the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  the  Office  of  the 
Assistant  Secretary  of  Defense  for  Readiness  had  instituted  key 
systemic  processes  to  help  ensure  that  the  data  entered  in  the 
Execution  Management  System  produce  quality  information — that  is, 
information  that  is  appropriate,  current,  complete,  accurate, 
accessible,  and  timely.  Standards  for  Internal  Control  in  the  Federal 
Government  states  that  a  variety  of  control  activities  should  be  used 
for  information  systems  to  support  the  completeness,  accuracy,  and 
validity  of  information  processing,  and  the  production  of  quality 
information.^^  In  addition,  management  should  evaluate  information 
processing  to  ensure  that  it  is  complete,  accurate,  and  valid.  Further, 
these  standards  state  that  appropriate  documentation  of  transactions 
should  be  readily  available  for  examination.  Using  these  internal 
controls  could  reduce  to  an  acceptable  level  the  risk  that  a  significant 
mistake  could  occur  and  remain  undetected  and  uncorrected. 
Individuals  from  all  four  of  the  combatant  commands  we  visited  stated 
that  only  one  person  at  their  combatant  command  was  responsible  for 
entering  data  into  the  Execution  Management  System  for  their 
respective  command  and  that,  although  they  believed  their  entries 
were  reliable,  no  quality  assurance  oversight  was  conducted  on  their 
work.  An  official  from  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Readiness  stated  that  periodic  reviews  are  conducted  on  data 
entered  in  the  Execution  Management  System  but  that  these  reviews 
are  mainly  focused  on  the  execution  rates  of  funds  and  not  on 
whether  the  data  entered  produces  quality  information.  Further, 


2^GAO-14-704G,  paragraphs  10.01  through  10.03,  11.02  through  11.14,  12.02  through 
12.05,  13.02  through  13.06. 
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according  to  an  official  from  the  Office  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff,  the  checks  they  perform  on  the  data  entered  in  the 
Execution  Management  System  are  similar  to  those  conducted  by  the 
Office  of  the  Assistant  Secretary  of  Defense  for  Readiness,  in  that 
they  are  focused  on  whether  or  not  monthly  expenditures  have  been 
entered  into  the  system  in  order  to  ensure  that  monthly  benchmarks 
are  met  and  less  on  whether  or  not  the  data  entered  produces  quality 
information.  According  to  officials  from  two  of  the  four  combatant 
commands  we  visited,  sometimes  they  receive  phone  calls  from  the 
Cffice  of  the  Chairman  of  the  Joint  Chiefs  of  Staff  or  the  Cffice  of  the 
Assistant  Secretary  of  Defense  for  Readiness  to  validate  certain  data 
entries  because  they  seemed  erroneous  based  on  an  informal  review. 
However,  no  officials  at  the  combatant  commands  we  visited,  the 
Cffice  of  the  Chairman  for  the  Joint  Chiefs  of  Staff,  or  the  Cffice  of  the 
Assistant  Secretary  of  Defense  for  Readiness  could  demonstrate 
systemic  processes  for  ensuring  that  the  Execution  Management 
System  produced  quality  information.  The  absence  of  quality 
assurance  processes  can  affect  the  quality  of  the  information 
produced  by  the  system  that  DCD  uses  to  determine  its  most  recent 
execution  rates  and  defend  the  Joint  Exercise  Program’s  budget. 

As  previously  discussed,  the  combatant  commands  are  not  following 
guidance  requiring  them  to  upload  supporting  documentation,  and  DCD 
lacks  effective  internal  controls  to  help  ensure  the  reliability  of  the  data  in 
the  system.  DCD  officials  acknowledged  the  issues  we  identified 
regarding  inadequate  supporting  documentation  and  data  reliability  within 
the  Execution  Management  System.  A  senior  DCD  official  from  the  Cffice 
of  the  Assistant  Secretary  of  Defense,  Readiness  (Resources)  stated  that 
DCD  plans  to  address  these  control  weaknesses  with  respect  to  the 
Combatant  Commander  Exercise  Engagement  and  Training 
Transformation  account  for  the  Joint  Exercise  Program  as  part  of  its 
implementation  of  DCD’s  FIAR  Guidance  beginning  in  fiscal  year  2018  to 
ensure  that  the  account  is  audit  ready. DCD  established  the  FIAR  Plan 
as  its  strategic  plan  and  management  tool  for  guiding,  monitoring,  and 
reporting  on  the  department’s  ongoing  financial  management 
improvement  efforts  and  for  communicating  the  department’s  approach  to 
addressing  its  financial  management  weaknesses  and  achieving  financial 
statement  audit  readiness.  To  implement  the  FIAR  Plan,  the  DCD 


As  defined  in  the  FIAR  Guidance,  an  audit-ready  entity  is  one  that  has 
strengthened  internal  controls  and  improved  financial  practices, 
processes,  and  systems  so  that  it  has  reasonable  confidence  the 
resulting  information  can  undergo  an  audit  by  an  independent  auditor. 
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Comptroller  issued  the  FIAR  Guidance,  which  provides  a  standard 
methodology  for  DOD  components  to  follow  to  assess  their  financial 
management  processes  and  controls  and  to  develop  and  implement 
financial  improvement  plans.  These  plans,  in  turn,  are  intended  to  provide 
a  framework  for  planning,  executing,  and  tracking  essential  steps  and 
related  supporting  documentation  needed  to  achieve  auditability.  We 
believe  that  if  DOD  appropriately  follows  the  steps  outlined  in  FIAR 
guidance  when  executing  the  Combatant  Commander  Exercise 
Engagement  and  Training  Transformation  account,  it  may  help  improve 
the  quality  of  funds  execution  data  from  this  account  and  make  the 
account  audit  ready. 

However,  as  previously  stated,  FIAR  guidance  will  not  be  implemented 
with  the  Joint  Exercise  Program  until  fiscal  year  2018  and  the 
effectiveness  of  the  guidance  cannot  be  fully  determined  until  after  that 
time.  In  the  meanwhile,  according  to  a  senior  DOD  official,  DOD  plans  to 
continue  using  the  Execution  Management  System  which  is  intended  to 
capture  the  most  recent  data  for  the  Joint  Exercise  Program  and  inform 
management  decision-making  regarding  joint  exercise  investments. 
Without  ensuring  the  required  supporting  documentation  is  uploaded  and 
implementing  effective  internal  controls  to  ensure  that  data  entered  in  the 
Execution  Management  System  produces  quality  information,  DOD  and 
other  key  decision  makers  may  not  have  the  correct  financial  execution 
information  to  defend  the  Joint  Exercise  Program’s  budget. 


Conclusions 


DOD  has  developed  a  body  of  guidance  for  the  Joint  Exercise  Program. 

In  addition,  DOD  has  implemented  an  approach  to  develop  performance 
measures  to  assess  the  effectiveness  of  the  Joint  Exercise  Program. 
Further,  DOD  uses  JTIMS  and  the  Execution  Management  System  to 
manage  the  Joint  Exercise  Program.  JTIMS  is  the  system  of  record  for 
executing  the  Joint  Exercise  Program  and  the  Office  of  the  Chairman  of 
the  Joint  Chiefs  of  Staff  officials  developed  a  user  guide  intended  to  help 
bring  more  standardization  to  the  system,  thereby  making  the  information 
more  useful  to  other  combatant  commands.  The  Execution  Management 
System  is  used  to  oversee  and  report  on  most  recent  execution 
performance  data  for  Joint  Exercise  Program  funding.  However,  not  all  of 
the  combatant  commands  were  following  guidance  requiring  them  to 
upload  supporting  documentation,  making  it  difficult  for  DOD  to  have 
oversight  on  expenditures  for  the  Joint  Exercise  Program.  Finally,  DOD 
and  the  combatant  commands  lack  systemic  processes  for  ensuring  that 
the  Execution  Management  System  produces  quality  information.  Without 
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ensuring  that  supporting  documentation  is  uploaded  and  implementing 
effective  internal  controls  to  ensure  the  completeness  and  accuracy  of 
financial  information  captured  for  the  Joint  Exercise  Program,  DOD  and 
other  key  decision  makers  may  not  have  the  correct  financial  information 
to  defend  the  Joint  Exercise  Program’s  budget. 


RprnmmpnHatinnci  for  ensure  quality  financial  execution  information  is  available  to 

r^euuriunenuduunb  lur  Exercise  Program,  we  recommend  that  the  Secretary  of 

Executive  Action  Defense  direct  the  Office  of  the  Assistant  Secretary  of  Defense  for 

Readiness  to  take  the  following  two  actions: 


Agency  Comments 
and  Our  Evaluation 


•  direct  the  combatant  commanders  to  take  steps  to  comply  with  current 
Execution  Management  System  guidance  to  upload  supporting 
documentation  that  is  reconcilable  to  funds  executed  from  the 
Combatant  Commanders  Exercise  Engagement  and  Training 
Transformation  account;  and 

•  as  the  department  implements  financial  improvement  plans  in 
accordance  with  the  FIAR  guidance,  it  should  include  specific  internal 
control  steps  and  procedures  to  address  and  ensure  the 
completeness  and  accuracy  of  information  captured  for  the  Joint 
Exercise  Program’s  Combatant  Commanders  Exercise  Engagement 
and  Training  Transformation  account. 

We  provided  a  draft  of  this  report  to  DOD  for  review  and  comment.  In  its 
written  comments,  which  are  summarized  below  and  reprinted  in 
appendix  IV,  DOD  partially  concurred  with  both  recommendations.  DOD 
also  provided  technical  comments,  which  we  incorporated  as  appropriate. 

DOD  partially  concurred  with  our  recommendation  to  direct  the  combatant 
commanders  to  take  steps  to  comply  with  current  Execution  Management 
System  guidance  to  upload  supporting  documentation  that  is  reconcilable 
to  funds  executed  from  the  Combatant  Commanders  Exercise 
Engagement  and  Training  Transformation  account.  In  its  comments,  DOD 
stated  that  the  Execution  Management  System  is  not  a  system  of  record 
but  rather  a  “desk-side”  support  tool  that  relies  on  manual  inputs  and 
uploads  and  that  the  reconciliation  of  obligation  and  execution  related  to 
Joint  Exercise  Program  funding  occurs  elsewhere.  DOD  further  noted  that 
the  Office  of  the  Assistant  Secretary  of  Defense  for  Readiness  (OASD(R)) 
issued  guidance  and  routinely  reinforces  the  best  practices  use  of  the 
Execution  Management  System  tool  to  ensure  it  produces  quality 
information.  Lastly,  DOD  noted  in  its  comments  that  that  it  may  not 
continue  using  the  Execution  Management  System  beyond  fiscal  year 
2017. 
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We  recognize  that  the  Execution  Management  System  is  not  a  system  of 
record.  However,  as  we  also  note  in  the  report,  it  is  a  tool  used  by  DOD  to 
make  decisions  regarding  the  Joint  Exercise  Program  because  the 
Defense  Finance  and  Accounting  Services  (DFAS)  Accounting  Report 
Monthly  1002,  the  system  of  record,  lags  behind.  Additionally,  we 
acknowledge  that  DOD  issued  guidance  for  the  Execution  Management 
System  in  April  2016,  but  our  work  found  that  the  guidance  was  not 
routinely  reinforced.  For  example,  as  we  identified  in  our  report,  two  of  the 
four  combatant  commands  we  visited  had  not,  in  fact,  uploaded 
supporting  documentation,  as  required  by  the  Execution  Management 
System  guidance,  for  fiscal  years  2013-2016.  While  the  Execution 
Management  System  may  not  be  funded  beyond  fiscal  year  2017,  we 
continue  to  believe  that  as  long  as  the  Execution  Management  System 
remains  in  use  and  for  the  reasons  discussed  in  the  report,  combatant 
commanders  should  take  the  necessary  steps  to  comply  with  existing 
guidance  that  requires  the  uploading  of  supporting  documentation  into  the 
Execution  Management  System  so  that  when  DOD  managers  make 
decisions  regarding  the  Joint  Exercise  Program  funding,  they  use 
information  from  a  financial  data  system  that  is  reconcilable  and 
auditable. 

DOD  partially  concurred  with  our  recommendation  that  states  that  as  the 
department  implements  financial  improvement  plans  in  accordance  with 
the  FIAR  guidance,  it  should  include  specific  internal  control  steps  and 
procedures  to  address  and  ensure  the  completeness  and  accuracy  of 
information  captured  for  the  Joint  Exercise  Program’s  Combatant 
Commanders  Exercise  Engagement  and  Training  Transformation 
account.  In  its  comments,  DOD  described  OASD(R)  as  having  a 
supporting  role  in  the  execution  of  FIAR  plans,  which  are  implemented  by 
Washington  Headquarters  Services  and  the  Office  of  Secretary  of 
Defense-Comptroller,  and  that  these  agencies  provide  specific  internal 
controls,  processes  and  procedures  for  ensuring  completeness  and 
accuracy  of  obligation  and  execution  data.  DOD  also  stated  that  the 
Execution  Management  System  is  not  a  component  of  FIAR  and  may  not 
be  funded  after  fiscal  year  2017,  and  that  moving  toward  audit  readiness, 
necessary  steps  and  procedures  will  be  put  into  place  to  strengthen 
auditability. 

As  we  stated  in  the  report,  the  FIAR  guidance  provides  a  standard 
methodology  and  framework  for  assessing  and  developing  a  system  of 
internal  controls  to  achieve  auditability.  However,  as  we  recommended, 
DOD  still  needs  to  implement  specific  internal  control  steps  and 
procedures  as  it  implements  this  guidance  to  ensure  the  completeness 
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and  accuracy  of  the  Joint  Exercise  Program’s  Combatant  Commanders 
Exercise  Engagement  and  Training  Transformation  account’s  financial 
information.  Further,  as  we  reported  the  FIAR  guidance  will  not  be 
implemented  in  the  Joint  Exercise  Program  until  fiscal  year  2018  and  the 
effectiveness  of  the  guidance  cannot  be  fully  determined  until  after  that 
time.  Accordingly,  we  continue  to  believe  that  the  recommendation 
remains  valid. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees,  the  Secretary  of  Defense;  the  Under  Secretary  of  Defense 
for  Personnel  and  Readiness;  the  Chairman  of  the  Joint  Chiefs  of  Staff; 
and  the  Commanders  of  U.S.  Northern  Command,  U.S.  Pacific 
Command,  U.S.  Strategic  Command,  and  U.S.  Transportation  Command. 
In  addition,  the  report  is  available  at  no  charge  on  the  GAO  website  at 
http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
me  at  (202)  512-5431  or  russellc@gao.gov.  Contact  points  for  our  Offices 
of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  Key  contributors  to  this  report  are  listed  in  appendix  V. 


Director 

Defense  Capabilities  and  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
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This  report  (1)  describes  guidance  the  Department  of  Defense  (DOD)  has 
deveioped  for  its  Joint  Exercise  Program  and  DOD’s  approach  to  assess 
the  effectiveness  of  the  program  and  (2)  evaiuates  the  extent  to  which 
DOD  uses  two  key  information  systems — the  Joint  Training  information 
Management  System  (JTiMS)  and  the  Execution  Management  System-to 
manage  the  Joint  Exercise  Program. 

To  describe  the  guidance  that  the  Department  of  Defense  has  deveioped 
for  its  Joint  Exercise  Program  and  DOD’s  approach  to  assess  the 
effectiveness  of  the  program,  we  reviewed  guidance  deveioped  by  the 
Office  of  the  Assistant  Secretary  of  Defense  for  Readiness,  the  Office  of 
the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  the  combatant  commands 
for  joint  exercises  and  other  joint  training  events  funded  by  the  Combatant 
Commanders  Exercise  Engagement  and  Training  Transformation 
account.  We  focused  on  funding  from  the  Combatant  Commanders 
Exercise  Engagement  and  Training  Transformation  account  because 
officiais  from  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Readiness  and  the  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff 
informed  us  that  this  account  funded  exercises  in  the  Joint  Exercise 
Program.  Specificaiiy,  we  reviewed  the  foiiowing  guidance  from  the 
foiiowing  DOD  entities: 


Office  of  the  Assistant 
Secretary  of  Defense 
for  Readiness 

Office  of  the 
Chairman  of  the  Joint 
Chiefs  of  Staff 


DODD  1322.18,  Miiitary  Training  (January  13,  2009) 


CJCSi  3500.01  H,  Joint  Training  Poiicy  for  the  Armed  Forces  of  the 
United  States  (Aprii  25,  2014) 

CJCSM  3150. 25A,  Joint  Lessons  Learned  Program  (September  12, 
2014) 

CJCSN  3500.01 , 2015-2018  Chairman’s  Joint  Training  Guidance 
(October  30,  2014) 

CJCSM  3500. 03E,  Joint  T raining  Manuai  for  the  Armed  Forces  of  the 
United  States  (Aprii  20,  2015) 

CJCS  Guide  3501 ,  The  Joint  T raining  System:  A  Guide  for  Senior 
Leaders  (May  5,  2015) 

CJCSM  351 1 .01 H,  Joint  Training  Resources  for  the  Armed  Forces  of 
the  United  States  (May  26,  2015) 
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•  CJCSI  3150.25F,  Joint  Lessons  Learned  Program  (June  26,  2015) 

We  compared  guidance  issued  by  the  Office  of  the  Assistant  Secretary  of 
Defense  for  Readiness  with  the  guidance  issued  by  the  Office  of  the 
Chairman  of  the  Joint  Chiefs  of  Staff  to  identify  similarities  and 
differences.  Additionally,  we  reviewed  the  following  implementation 
guidance  from  the  four  combatant  commands  that  we  visited: 

U.S.  Northern 

•  NORAD  and  NORTHCOM,  Joint  Training  System  (JTS)  Handbook 
(May  1, 2013) 

Command 

•  NORAD  and  NORTHCOM  Instruction  16-166,  Lessons  Learned 
Program  and  Corrective  Action  Program  (September  19,  2013) 

.  NORAD  and  NORTHCOM,  JTPs,  (December  10,  2015) 

U.S.  Pacific 

■ 

•  PACOM  Instruction  0509.1 ,  Joint  Lessons  Learned  and  Issue 

Resolution  Program  (April  7,  2010) 

Command 

•  PACOM  Instruction  0508.12,  Joint  Training  Enterprise  in  U.S.  Pacific 
Command  (October  15,  2012) 

U.S.  Strategic 
Command 

•  SI  508-09,  Exercise  Program  (May  3,  2013) 

•  SI  509-01,  After  Action,  Issue  Solution/Resolution  and  Lessons 
Learned  Program,  October  14,  2015) 

•  SI  508-03,  JTIMS  Procedures  (November  8,  2015) 

U.S.  Transportation 

•  USTRANSCOM  Pamphlet  38-1 ,  Organization  and  Functions  (May  1 , 
2008) 

Command 

•  USTRANSCOM  Instruction  36-13,  Training,  Education,  and 
Professional  Development  Program  (May  23,  2013) 

•  USTRANSCOM  Instruction  36-36,  Joint  Training  and  Education 
Program  (August  29,  2014) 

•  USTRANSCOM  Instruction  10-14,  Joint  Lessons  Learned  Program 

(November  9,  2015) 

We  judgmentally  selected  these  combatant  commands  for  our  site  visits 
to  achieve  a  mix  of  geographical  and  functional  commands,  as  well  as  the 
funds  that  had  been  apportioned  to  the  combatant  commands  in  fiscal 
year  2016  from  the  Combatant  Commanders  Exercise  Engagement  and 
Training  Transformation  account,  size  of  command,  and  location.  We 
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reviewed  DOD’s  approach  to  make  performance  measures  specific, 
measurable,  achievable,  realistic,  and  time-phased  (commonly  referred  to 
as  the  SMART  rubric)  to  assess  the  return  on  investment  for  the  Joint 
Exercise  Program.  In  addition,  we  reviewed  performance  measures 
reportedly  used  to  assess  the  ability  of  the  training  audience  to 
accomplish  training  objectives  for  exercises,  as  well  as  measures  used  to 
assess  the  return  on  investment  for  conducting  an  exercise.  We  also 
reviewed  performance  documentation  and  information  captured  in  JTIMS, 
as  well  as  a  nongeneralizable  sample  of  commander  summary  reports  or 
after-action  reports  from  seven  combatant  command  joint  exercises  to 
understand  the  content  of  these  reports.  Finally,  we  interviewed  senior 
officials  from  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Readiness,  including  the  Joint  Assessment  and  Enabling  Capability 
office,  the  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff,  as  well  as 
officials  from  the  four  selected  combatant  commands.  We  did  not  review 
or  evaluate  the  quality  of  any  assessments  that  DOD  has  conducted  for 
its  joint  exercises.  We  reviewed  the  assessments  to  the  extent  that  it  was 
possible  to  ensure  that  an  assessment  process  existed. 

To  evaluate  DOD’s  use  of  JTIMS''  and  the  Execution  Management 
System  to  manage  the  Joint  Exercise  Program,  we  reviewed  guidance  for 
JTIMS  and  the  Execution  Management  System.  In  addition,  we  observed 
data  associated  with  a  nongeneralizable  sample  of  joint  exercises 
maintained  in  JTIMS.  We  also  reviewed  and  analyzed  a  nongeneralizable 
sample  of  cumulative  financial  data  and  supporting  documentation,  if  any, 
entered  by  combatant  command  users  in  the  Execution  Management 
System  for  the  Joint  Exercise  Program  during  fiscal  years  2013-16  to 
examine  the  internal  controls  that  were  in  place. ^  We  reviewed  a 
nongeneralizable  sample  of  supporting  documentation  uploaded  into  the 
Execution  Management  System  for  fiscal  years  2014  through  2016  to 
make  a  determination  about  compliance  with  guidance  issued  by  DOD  for 
the  Execution  Management  System  and  the  Standards  for  Internal 


^The  Joint  Training  Information  Management  System  is  the  system  of  record  used  by  the 
combatant  commands  to  design,  plan,  prepare,  execute,  and  evaluate  exercises. 

^According  to  a  DOD  official,  the  Combatant  Commanders  Exercise  Engagement  and 
Training  Transformation  enterprise  began  using  the  Execution  Management  System  in 
2008.  The  Execution  Management  System  only  allows  visibility  of  data  for  the  years  when 
users  are  able  to  input  activity.  Procurement  dollars  have  a  3-year  window  for  obligations 
and  a  fourth  year  for  expenditures.  Therefore,  the  earliest  year  that  is  visible  for  users  to 
input  activity  into  the  system  is  fiscal  year  2013  since  procurement  funds  for  that  year 
were  available  to  obligate  through  fiscal  year  201 5  and  would  be  expended  throughout 
fiscal  year  2016. 
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Control  in  the  Federal  Government.^  Further,  we  compared  individual 
transactions  reported  in  the  supporting  documentation  with  the 
corresponding  cumulative  data  entered  into  the  system.  We  also 
reviewed  the  FIAR  plan —  DOD’s  strategic  plan  and  management  tool  for 
guiding,  monitoring,  and  reporting  on  the  department’s  ongoing  financial 
management  improvement  efforts  and  for  communicating  the 
department’s  approach  to  addressing  its  financial  management 
weaknesses  and  achieving  financial  statement  audit  readiness —  and 
guidance.  Additionally,  we  spoke  with  cognizant  officials  from  the  four 
combatant  commands  we  visited,  the  Office  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff,  and  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Readiness  about  the  systems  used  to  execute  and  manage  the  Joint 
Exercise  Program.  Further,  we  attended  sessions  at  the  3-day  Annual 
Review  for  the  Combatant  Commanders  Exercise  Engagement  and 
Training  Transformation  Enterprise  on  the  budget  for  fiscal  years  2018 
through  2022.  We  attended  sessions  that  were  most  pertinent  to  this 
engagement.  For  example,  since  the  services  were  not  included  in  the 
scope  of  our  review,  we  did  not  attend  their  sessions. 

We  conducted  site  visits  to  collect  testimonial  and  documentary  evidence 
about  DOD’s  Joint  Exercise  Program  at  the  following  locations: 

•  Cost  Assessment  and  Program  Evaluation  Office,  Arlington,  Virginia 

•  Force  Readiness  and  Training  in  the  Office  of  the  Assistant  Secretary 
of  Defense  for  Readiness,  Arlington,  Virginia 

•  Joint  Assessment  and  Enabling  Capability  office,  Alexandria,  Virginia 

•  Joint  Staff  (J7),  Arlington,  Virginia 

•  Joint  Staff  (J7)  Suffolk,  Virginia 

•  U.S.  Northern  Command,  Peterson  Air  Force  Base,  Colorado 

•  U.S.  Pacific  Command,  Camp  H.  M.  Smith,  Hawaii 

•  U.S.  Strategic  Command,  Offutt  Air  Force  Base,  Nebraska 

•  U.S.  Transportation  Command,  Scott  Air  Force  Base,  Illinois 

We  conducted  this  performance  audit  from  June  2015  to  February  2017 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 


^GAO-14-704G. 
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sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 
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Guidance  from  the  Office  of  the  Chairman  of  the  Joint  Chiefs  of  Staff 
outlines  the  process  that  is  used  by  the  combatant  commands  to  develop 
joint  training  programs,  plan  and  execute  joint  training,  and  assess 
training  for  the  Department  of  Defense’s  (DOD)  Joint  Exercise  Program.'' 
This  process,  referred  to  in  the  guidance  as  the  Joint  Training  System, ^  is 
characterized  as  an  integrated,  requirements-based,  four-phased 
methodology  used  to  align  the  Joint  Training  Strategy  with  assigned 
missions  to  produce  trained  and  ready  individuals,  staff,  and  units. 
According  to  the  guidance,  the  Joint  Training  System  has  four  phases 
through  which  the  combatant  commands  execute  the  Joint  Exercise 
Program. 

•  Phase  I — Requirements.  During  this  phase,  an  ordered  listing  of 
tasks^  is  developed  describing  the  armed  force’s  ability  to  perform 
activities  or  processes  that  combatant  commanders  require  to  execute 
their  assigned  missions.  This  listing  is  referred  to  as  the  Universal 
Joint  Task  List  and  it  provides  a  common  language  to  describe 
warfighting  requirements  for  combatant  commanders.  From  this  list, 
the  most  essential''  mission  capability  tasks — mission-essential 
tasks — are  identified  by  the  combatant  commander.  Using  the 
commander’s  criteria,  mission-essential  tasks  are  prioritized  to  form 
the  Joint/Agency  Mission-Essential  Task  List.  In  addition  to  combatant 
commanders’  priority,  key  documents  pertinent  to  U.S.  national 
strategy,  such  as  the  Unified  Command  Plan,  Guidance  for 
Employment  of  the  Force,  and  other  joint  doctrine,  are  analyzed  to 
determine  the  most  essential  mission  capability  requirements  for  the 
combatant  command.  The  Joint/Agency  Mission-Essential  Task  List 
provides  the  foundation  for  deriving  joint  training  requirements  used  to 
develop  Joint  Training  Plans  and  training  and  exercise  inputs  to 
theater  campaign  plans.  Training  requirements  are  derived  from 
training  proficiency  assessments,  mission  training  assessments,  and 


^Chairman  of  the  Joint  Chiefs  of  Staff,  CJCSM  3500. 03E,  Joint  Training  Manuai  for  the 
Armed  Forces  of  the  United  States,  (April  20,  2015) 

^According  to  guidance  CJCSM  3500. 03E,  Joint  Training  Manuai  for  the  Armed  Forces  of 
the  United  States,  (April  20,  2015),  although  the  Joint  Training  System  is  a  deliberate 
process  in  concept,  the  process  is  flexible  in  execution. 

^Tasks  are  organized  by  levels  of  war  (strategic,  operational,  or  tactical). 

''The  definition  or  criteria  for  what  makes  a  task  “essential”  is  specific  to  each  mission.  The 
core  criteria  for  essentiality  are  whether  or  not  the  objective  can  be  attained  and  the 
mission,  accomplished  without  the  task  being  performed  to  the  set  standard. 
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lessons  learned  that  result  from  the  Phase  IV  (Assessment)  of  the 
Joint  Training  System. 

•  Phase  II — Plans.  The  plans  phase  is  initiated  by  conducting  an 
assessment  of  current  capability  against  the  Joint  Mission-Essential 
Task  List  and  relevant  lessons  learned  to  identify  gaps  in  training.  To 
address  those  gaps,  the  Joint  Training  Plan  is  established  and 
identifies  who  is  to  be  trained;  what  they  will  be  trained  in;  what  the 
training  objectives  are;  and  when,  where,  and  how  the  training  will 
occur.  Joint  Training  Plans,  along  with  training  and  exercise  inputs 
into  theater  campaign  plans,  are  developed,  coordinated,  and 
published  in  the  Joint  Training  Information  Management  System 
(JTIMS)  to  identify  a  commander’s  training  guidance,  audiences, 
objectives,  events,  and  support  resources,  and  to  identify  the 
coordination  needed  to  attain  the  required  levels  of  training 
proficiency. 

•  Phase  III — Execution.  During  this  phase,  events  planned  in  the  Joint 
Training  Plan  are  conducted  and  the  training  audience’s  performance 
objectives  are  observed  and  evaluated.  Joint  training  events  are 
developed  and  executed  using  the  five-stage  Joint  Event  Life  Cycle® 
methodology  captured  and  reviewed  in  JTIMS.  Task  Performance 
Observations — which  identify  whether  the  training  audience  achieved 
the  stated  level  of  performance  to  the  standards  specified  in  the 
training  objectives — and  the  Training  Proficiency  Evaluations®  for 
each  training  objective  associated  with  the  training  event  are  also 
captured  in  JTIMS.  Further,  facilitated  after-action  reports  are 
developed  to  highlight  potential  issues  or  best  practices  to  support  the 
assessments  in  Phase  IV  (Assessment).  Validated  observations  from 
the  training  event  are  exported  into  JTIMS. 

•  Phase  IV — Assessment.  During  this  phase,  leadership  within  the 
combatant  command  determines  which  organizations  within  the 
command  are  able  to  perform  at  the  level  required  to  meet  the  task 
standards  and  which  missions  the  command  is  trained  to  accomplish. 
Assessments  are  a  commander’s  responsibility.  To  complete  Task 
Performance  Assessments  for  each  task,  commanders  consider  Task 


®The  Joint  Event  Life  Cycle  comprises  the  design,  planning,  preparation,  execution, 
evaluation,  and  reporting  stages  required  to  successfully  execute  a  training  event.  The 
Joint  Event  Life  Cycle  is  referred  to  as  a  cycle  within  a  cycle. 

^Training  proficiency  evaluations  identify  how  well  the  training  audience  performed  training 
objectives  during  an  observed  training  event  against  the  standards  of  performance  for  that 
objective. 
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Performance  Evaluations,  lesson  learned,  and  personal  observations 
of  the  joint  training  exercise.  An  assessment  ranking  of  trained, 
partially  trained,  or  untrained  is  assigned  to  each  task  listed  under  a 
training  objective.  JTIMS  supports  the  assessment  of  joint  training  by 
automating  the  ability  of  joint  organizations  to  produce  Task 
Performance  Assessments.  The  Task  Performance  Assessments  are 
analyzed  to  create  the  mission  training  assessment  that  is  provided  to 
a  combatant  commander  on  a  monthly  basis.  The  mission  training 
assessment  is  on  how  well  the  command  can  execute  its  assigned 
missions.  These  training  assessments  provide  input  into  the  next 
training  cycle.  Lessons  learned,  after-action  reports,  and  issues 
requiring  resolution  outside  of  the  command  are  identified  during  this 
phase. 
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Combatant  command  officials  we  visited  stated  that,  in  accordance  with 
guidance  from  the  Chairman  of  the  Joint  Chiefs  of  Staff,  ^  they  used  the 
Joint  Training  System  as  the  process  for  conducting  training  assessments 
of  individual  joint  exercises  to  determine  each  command’s  overall 
readiness  to  perform  command  missions.  These  assessments  occur 
during  Phases  III  (Execution)  and  IV  (Assessment)  of  the  Joint  Training 
System.  During  Phase  III,  for  example,  command  trainers  collect  task 
performance  observations  for  each  training  objective  identified  in  the  Joint 
Training  Information  Management  System  (JTIMS).  These  task 
performance  observations  identify  whether  the  individuals  and  units 
participating  in  the  training  exercise  achieved  the  level  of  performance 
stated  in  standards  specified  in  the  training  objectives.  Training 
proficiency  evaluations  are  conducted  for  each  training  objective 
associated  with  the  exercise.  During  Phase  IV,  combatant  commanders 
consider  the  proficiency  evaluations,  as  well  as  after-action  and 
commander  summary  reports,  to  determine  a  combatant  command’s 
ability  to  perform  assigned  missions  at  the  minimum  acceptable  level 
under  a  specified  set  of  conditions. 

According  to  an  official  from  the  Joint  Assessment  and  Enabling 
Capability  office,  a  subordinate  office  to  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Readiness  that  provides  strategic-  level 
assessments  of  joint  training  and  joint  training  enablers  throughout  the 
Department  of  Defense  (DOD),  including  to  combatant  commands, 
performance  measures  that  are  specific,  measurable,  achievable, 
realistic,  and  time-phased  (commonly  referred  to  as  the  SMART  rubric) 
are  used  to  assess  the  Joint  Exercise  Program.  DOD  officials  stated  that 
developing  performance  measures  for  joint  exercises  has  not  been  an 
easy  task  and  that  they  are  constantly  working  to  improve  their 
performance  measures.  Specifically,  in  an  effort  to  develop,  improve,  and 
provide  quality  assurance  for  specific  performance  measures,  the  Joint 
Assessment  and  Enabling  Capability  office  works  with  the  combatant 
commands  to  ensure  that  they  are  using  the  right  measures  to  evaluate 
the  training  audience’s  ability  to  perform  tasks  to  specific  standards.  In 
addition,  the  Joint  Assessment  and  Enabling  Capability  office  hosts 
monthly  meetings  with  combatant  command  stakeholders  to  discuss 
assessment  topics,  including  performance  measures.  Additionally,  the 
Joint  Assessment  and  Enabling  Capability  office  hosts  at  least  one 


^  Chairman  of  the  Joint  Chiefs  of  Staff,  CJCSI  3500.01  H,  Joint  Training  Poiicy  for  the 
Armed  Forces  of  the  United  States,  (April  25,  2014). 
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working  group  meeting  at  the  annuai  woridwide  joint  training  conference 
to  conduct  face-to-face  discussions  and  reviews  of  assessment-reiated 
tasks  for  joint  training.  According  to  an  annuai  report  from  the  Joint  Staff 
Director  for  Joint  Force  Deveiopment,  the  Joint  Assessment  and  Enabiing 
Capabiiity  office  is  avaiiabie  to  assist  combatant  command  stakehoiders 
with  assessment-reiated  tasks  for  the  Joint  Exercise  Program,  as 
requested. 
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OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 

4000  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-4000 


READINESS 


February  13,  2017 
Mr.  Cary  Russell 

Director,  Defense  Capabilities  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington  DC  20548 

Dear  Mr.  Russell: 

This  is  the  Department  of  Defense  (DoD)  amended  response  to  the  Government 
Accountability  Office  (GAO)  Draft  Report  GAO-17-7,  “JOINT  EXERCISE  PROGRAM:  DOD 
Needs  to  Take  Steps  to  Improve  the  Quality  of  Funding  Execution  Data,”  dated  Eebruary  1,  2017 
(GAO  Code  100177). 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report.  We  partially 
concur  with  Recommendation  1  and  2.  The  Department’s  comments  on  the  GAO  draft 
recommendations  are  enclosed. 


I  can  be  reached  at  703-693-7495  orjohn.t.hastings4.civ@mail.mil. 

Sincerely, 


A  />-- 


John  T.  Hastings 

Acting  Principal  Deputy  Assistant  Secretary 
of  Defense  (Readiness) 


Attachments: 
As  stated 
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Appendix  IV:  Comments  from  the  Department 
of  Defense 


GAO  DRAFT  REPORT  DATED  FEBRUARY  1,  2017 
GAO-17-7  (GAO  CODE  100177) 

“JOINT  EXERCISE  PROGRAM:  DOD  NEEDS  TO  TAKE  STEPS  TO  IMPROVE  THE 
QUALITY  OE  EUNDING  EXECUTION  DATA” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


To  better  ensure  that  the  Execution  Management  System  produces  quality  information,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Office  of  the  Assistant  Secretary  of 
Defense  (Readiness)  to  take  the  following  actions: 

RECOMMENDATION  1:  direct  the  combatant  commanders  to  take  steps  to  comply  with 
current  Execution  Management  System  guidance  to  upload  supporting  documentation  that  is 
reconcilable  to  funds  executed  from  the  Combatant  Commanders  Exercise  Engagement  and 
Training  Transformation  account;  and 

DoD  RESPONSE;  Partially  concur.  The  Execution  Management  System  (EMS)  is  a  desk-side 
support  tool  provided  to  the  Enterprise  as  a  means  of  maintaining  a  program  “check  register.”  It 
is  not  a  system  of  record  and  relies  on  manual  inputs  and  uploads.  Obligation  and  execution  data 
occur  in  the  DoD  systems  of  record.  The  Enterprise  is  responsible  for  reconciling  obligation  and 
execution  of  Joint  Exercise  Program  funding  within  the  systems  of  record  to  ensure  the  accuracy 
of  the  Defense  Finance  and  Accounting  Services  (DFAS)  Accounting  Report  Monthly  1002 
(AR(M)  1002).  Defense- wide  account  reporting  requires  on-going  attention  and  reinforcement. 
EMS  provides  Enterprise  resource  managers  (within  the  Combatant  Commands)  a  tool  for 
capturing  detailed  financial  information;  and  assists  leaders  in  monitoring  spending.  As  noted  in 
the  report,  OASD(R)  issued  guidance  and  routinely  reinforces  the  best  practices  use  of  the  EMS 
tool  to  ensure  it  produces  quality  information.  This  oversight  is  ongoing. 

RECOMMENDATION  2:  as  the  department  implements  financial  improvement  plans  in 
accordance  withFIAR  guidance,  it  should  include  specific  internal  control  steps  and  procedures 
to  address  and  ensure  the  completeness  and  accuracy  of  information  captured  for  the  Joint 
Exercise  Program 's  Combatant  Commanders  Exercise  Engagement  and  Training 
Transformation  account. 

DoD  RESPONSE:  Partially  concur.  OASD(R)  provides  a  supporting  role  in  the  execution  of 
financial  improvement  audit  readiness  (FIAR)  plans  implemented  by  Washington  Headquarters 
Services  (WHS)  mid  OUSD-Comptroller.  Specific  internal  controls,  processes  and  procedures 
for  ensuring  completeness  and  accuracy  of  obligation  and  execution  data  m'e  provided  to  the 
Enterprise  funded  activities  by  the  aforementioned  agencies.  The  EMS  is  not  a  component  of 
FIAR  and  may  not  be  funded  after  FY17.  Moving  toward  audit  readiness,  necessary  steps  and 
procedures  will  be  put  into  place  to  strengthen  auditability.  OASD(R)  provides  an  oversight 
role,  reviewing  and  monitoring  program  execution  and  the  accuracy  of  financial  reporting  on  the 
DFAS  1002. 
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Appendix  V:  GAO  Contact  and  Staff 
Acknowledgments 


GAO  Contact 


Cary  B.  Russell,  (202)  512-5431  or  russellc@gao.gov 


Staff 

Acknowledgments 


Guy  A.  LoFaro,  Assistant  Director;  Patricia  Donahue;  Pamela  Nicole 
Harris;  Amie  Lesser;  Sabrina  Streagle;  Sonja  S.  Ware;  and  Cheryl  A. 
Weissman  made  key  contributions  to  this  report. 
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Related  GAO  Products 


Civil  Support:  DOD  Needs  to  Clarify  Its  Roles  and  Responsibilities  for 
Defense  Support  of  Civil  Authorities  during  Cyber  Incidents.  GAO-16-332. 
Washington,  D.C:  April,  4,  2016. 

Military  Base  Realignments  and  Closures:  More  Guidance  and 
Information  Needed  to  Take  Advantage  of  Opportunities  to  Consolidate 
Training.  GAO-16-45.  Washington,  D.C.:  February  18,  2016. 

Operational  Contract  Support:  Actions  Needed  to  Enhance  the  Collection, 
Integration,  and  Sharing  of  Lessons  Learned.  GAO-15-243.  Washington, 
D.C.:  March  16,  2015. 

Defense  Headquarters:  DOD  Needs  to  Reevaluate  Its  Approach  for 
Managing  Resources  Devoted  to  the  Functional  Combatant  Commands. 
GAO-14-439.  Washington,  D.C.:  June  26,  2014. 

Defense  Headquarters:  DOD  Needs  to  Periodically  Review  and  Improve 
Visibility  of  Combatant  Commands’  Resources.  GAO-1 3-293. 

Washington,  D.C.:  May  15,  2013. 

Defense  Management:  Perspectives  on  the  Involvement  of  the 
Combatant  Commands  in  the  Development  of  Joint  Requirements. 

GAO-1 1-527R.  Washington,  D.C.:  May  20,  201 1 . 

Homeland  Defense:  U.S.  Northern  Command  Has  a  Strong  Exercise 
Program,  but  Involvement  of  Interagency  Partners  and  States  Can  Be 
Improved.  GAO-09-849.  Washington,  D.C.:  September  9,  2009. 

National  Preparedness:  FEMA  Has  Made  Progress,  but  Needs  to 
Complete  and  Integrate  Planning,  Exercise,  and  Assessment  Efforts. 
GAO-09-369.  Washington,  D.C.:  April  30,  2009. 

Homeland  Defense:  Steps  Have  Been  Taken  to  Improve  U.S.  Northern 
Command’s  Coordination  with  States  and  the  National  Guard  Bureau,  but 
Gaps  Remain.  GAO-08-252.  Washington,  D.C.:  April  16,  2008. 

Homeland  Defense:  U.S.  Northern  Command  Has  Made  Progress  but 
Needs  to  Address  Force  Allocation,  Readiness  Tracking  Gaps,  and  Other 
Issues.  GAO-08-251.  Washington,  D.C.:  April  16,  2008. 

Homeland  Security:  Observations  on  DHS  and  FEMA  Efforts  to  Prepare 
for  and  Respond  to  Major  and  Catastrophic  Disasters  and  Address 
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Related  GAO  Products 


Related  Recommendations  and  Legislation.  GAO-07-835T.  Washington, 
D.C.:  May  15,  2007. 

Military  Training:  Management  Actions  Needed  to  Enhance  DOD’s 
Investment  in  the  Joint  National  Training  Capability.  GAO-06-802: 
Washington,  D.C.:  August  1 1 , 2006. 

Military  Training:  Actions  Needed  to  Enhance  DOD’s  Program  to 
Transform  Joint  Training.  GAO-05-548:  Washington,  D.C.:  June  21 , 2005. 


(100177) 
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GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card,  MasterCard, 

Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 

Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngc1@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 

Washington,  DC  20548 

Strategic  Planning  and 
External  Liaison 

James-Christian  Blockwood,  Managing  Director,  spel@gao.gov,  (202)  512-4707 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7814, 
Washington,  DC  20548 

Please  Print  on  Recycled  Paper. 

